Building Goal-Oriented Dialogue Systems with Situated Visual Context

Goal-oriented dialogue agents can comfortably utilize the conversational context and understand its users' goals. However, in visually driven user experiences, these are also required to make sense of screen order provide a proper interactive experience. In this paper, we propose novel multimodal framework where agent's next action their arguments derived jointly conditioned both on visual context. We demonstrate proposed approach via prototypical furniture shopping experience for virtual assistant.